1.2.1 Plan for today

• General aims and structure
• Expectations
• Pointers for exercises
• Quick DeclareDesign intro

1.2.2 Aims

• Deep understanding of key ideas in causal inference
• Transportable tools for understanding how to evaluate and improve design
• Applied skills for design and analysis
• Exposure to open science practices
• Deeper dive into some specific topics (see survey)

1.2.3 Syllabus and resources

• https://macartan.github.io/ci for all resources
• https://macartan.github.io/ci/syllabus.pdf
• Git repo https://github.com/macartan/ci
• Student survey: Please fill this out today.

1.2.4 The topics

Day 1: Intro

• 1.1 Course outline, tools
• 1.2 Introduction to DeclareDesign

Day 2: Causality

• 2.1 Fundamental problems and solutions
• 2.2 Inquiries and identification

1.2.5 The topics

Day 3: Estimation and Inference

• 3.1 Frequentist
• 3.2 Bayesian

Day 4:

• 4.1 Experimental Design (esp. assignment)
• 4.2 Design evaluation (esp. power)

Day 5:

• 5.1 Topics and techniques
• 5.2 Open science

1.2.8 Good coding rules

• https://bookdown.org/content/d1e53ac9-28ce-472f-bc2c-f499f18264a3/code.html
• https://www.r-bloggers.com/2018/09/r-code-best-practices/

1.2.9 Good coding rules

• Metadata first
• Call packages at the beginning: use pacman
• Put options at the top
• Call all data files once, at the top. Best to call directly from a public archive, when possible.
  
• Use functions and define them at the top: comment them; useful sometimes to illustrate what they do
• Replicate first, re-analyze second. Use sections.
• (For replications) Have subsections named after specific tables, figures or analyses

1.2.10 Aim

• First best: If someone has access to your .Rmd/.qmd file they can hit render or compile and the whole thing reproduces first time. So: Nothing local, everything relative: so please do not include hardcoded paths to your computer

• But: often you need ancillary files for data and code. That’s OK but aims should still be that with a self contained folder someone can open a main.Rmd file, hit compile and get everything. I usually have an input and an output subfolder.

1.2.11 Collaborative coding / writing

• Do not get in the business of passing attachments around
• Documents in some cloud: git, osf, Dropbox, Drive, Nextcloud
• General rule: only post non sensitive, non proprietary material
• Share self contained folders; folders contain a small set of live documents plus an archive. Old versions of documents are in archive. Only one version of the most recent document is in a main folder.
• Data is self contained folder (in) and is never edited directly
• Update to github frequently

2.2.1 Four elements of any research design

• Model: set of models of what causes what and how
• Inquiry: a question stated in terms of the model
• Data strategy: the set of procedures we use to gather information from the world (sampling, assignment, measurement)
• Answer strategy: how we summarize the data produced by the data strategy

2.2.2 Four elements of any research design

2.2.3 Declaration

Design declaration is telling the computer (and readers) what M, I, D, and A are.

2.2.4 Diagnosis

• Design diagnosis is figuring out how the design will perform under imagined conditions.

• Estimating “diagnosands” like power, bias, rmse, error rates, ethical harm, “amount learned”.

• Diagnosis takes account of model uncertainty: it aims to identify models for which the design works well and models for which it does not

2.2.5 Redesign

Redesign is the fine-tuning of features of the data- and answer strategies to understand how changing them affects the diagnosands

• Different sample sizes
• Different randomization procedures
• Different estimation strategies
• Implementation: effort into compliance versus more effort into sample size

2.2.6 Very often you have to simulate!

• Doing all this is often too hard to work out from rules of thumb or power calculators
• Specialized formulas exist for some diagnosands, but not all

2.3.1 Key commands for making a design

• declare_model()

• declare_inquiry()

• declare_sampling()

• declare_assignment()

• declare_measurement()

• declare_estimator()

and there are more declare_ functions!

2.3.2 Key commands for using a design

• draw_data(design)
• draw_estimands(design)
• draw_estimates(design)
• get_estimates(design, data)
• run_design(design), simulate_design(design)
• diagnose_design(design)
• redesign(design, N = 200)
• compare_designs(), compare_diagnoses()

2.3.3 Pipeable commands

design |> 
  redesign(N = c(200, 400)) |>
  diagnose_designs() |> 
  tidy() |> 
  ggplot(...) 

2.3.4 Cheat sheet

https://raw.githubusercontent.com/rstudio/cheatsheets/master/declaredesign.pdf

2.3.5 Other resources

• The website: https://declaredesign.org/
• The book: https://book.declaredesign.org
• The console: ?DeclareDesign

2.4.1 The simplest possible (diagnosable) design?

mean <- 0
simplest_design <- 
  declare_model(N = 100, Y = rnorm(N, mean)) +
  declare_inquiry(Q = mean) +
  declare_estimator(Y ~ 1)

• we draw 100 units from a standard normal distribution
• we define our inquiry as the population expectation
• we estimate the average using a regression with a constant term

2.4.2 The simplest possible design?

simplest_design <- 
  declare_model(N = 100, Y = rnorm(N, mean)) +
  declare_inquiry(Q = 0) +
  declare_estimator(Y ~ 1)

• This design has three steps, with steps connected by a +
• The design itself is just a list of steps and has class design

str(simplest_design)

List of 3
 $ model    :design_step:    declare_model(N = 100, Y = rnorm(N, mean)) 
 $ Q        :design_step:    declare_inquiry(Q = 0) 
 $ estimator:design_step:    declare_estimator(Y ~ 1) 
 - attr(*, "call")= language construct_design(steps = steps)
 - attr(*, "class")= chr [1:2] "design" "dd"

2.4.3 The simplest possible design? It’s a pipe

Each step is a function (or rather: a function that generates functions) and each function presupposes what is created by previous functions.

• The ordering of steps is quite important
• Most steps take the main data frame in and push the main dataframe out; this data frame normally builds up as you move along the pipe.

2.4.4 The simplest possible design? It’s a pipe

Each step is a function (or rather: a function that generates functions) and each function presupposes what is created by previous functions.

• The ordering of steps is quite important
• declare_estimator steps take the main data frame in and send out an estimator_df dataframe
• declare_inquiry steps take the main data frame in and send out an estimand_df dataframe.

2.4.5 The simplest possible design? It’s a pipe

• You can run these functions one at a time if you like.
• For instance the third step presupposes the data from the first step:

df <- simplest_design[[1]]()
A  <- simplest_design[[3]](df)

A |> kable(digits = 2) |> kable_styling(font_size = 20)

Estimand  <- simplest_design[[2]](df)

Estimand |> kable(digits = 2) |> kable_styling(font_size = 20)

2.4.6 The simplest possible design? Run it once

You can also just run through the whole design once by typing the name of the design:

simplest_design

Research design declaration summary

Step 1 (model): declare_model(N = 100, Y = rnorm(N, mean)) ---------------------

Step 2 (inquiry): declare_inquiry(Q = 0) ---------------------------------------

Step 3 (estimator): declare_estimator(Y ~ 1) -----------------------------------

Run of the design:

 inquiry estimand estimator        term estimate std.error statistic p.value
       Q        0 estimator (Intercept)  0.00249    0.0916    0.0272   0.978
 conf.low conf.high df outcome
   -0.179     0.184 99       Y

2.4.7 The simplest possible design? Run it again

Or by asking for a run of the design

one_run <- simplest_design |> run_design()
one_run |> kable(digits = 2) |> kable_styling(font_size = 18)

A single run creates data, calculates estimands (the answer to inquiries) and calculates estimates plus ancillary statistics.

2.4.8 The simplest possible design?: Simulation

Or by asking for a run of the design

some_runs <- simplest_design |> simulate_design(sims = 1000)
some_runs |> kable(digits = 2) |> kable_styling(font_size = 16)